The Wisdom of Crowds in Rank Ordering Problems

When averaging the estimates of individuals, the aggregate can often come surprisingly close to the true answer. We are interested in extending this “wisdom of crowds” phenomenon to more complex situations where a simple strategy like taking the mode or mean of responses is inappropriate, or might lead to bad predictions. We report the performance of individuals in a series of ordering tasks, where the goal is to reconstruct from memory the order of time-based events, or the magnitude of physical properties. We introduce a Bayesian version of a Thurstonian model that aggregates orderings across individuals, and compare it to heuristic aggregation techniques inspired by existing models of social choice and voting theory. The Bayesian model performs as well as the heuristics in reconstructing the true ordering, and has the advantage of being well calibrated, in the sense that it gives more confident responses the closer it is to the truth.